(1) Andrea Soltoggio and Jochen Steil, Solving the Distal Reward Problem with Rare Correlations, Neural Computation, 25, p940-978. 2013 http://www.mitpressjournals.org/doi/abs/10.1162/NECO_a_00419

How do animals establish associations between cues, actions and rewards when rewards come sometime later after the actions that caused them?

The research in the paper “Solving the Distal Reward Problem with Rare Correlations”(1) proposes a new type of computation based on the rarity of correlations. Some neural events that happen rarely are selected to leave a trace which can be detected later in time when a reward occurs. The study shows how the neural models is capable of classical and operant conditioning with delayed rewards.

The PDF of the paper is available for download.  Support material include the Matlab code to reproduce the experiments and a short video.

Download PDF of Solving the Distal Reward Problem with Rare Correlations

Download the Matlab code and support video

How the idea came about

The main idea was inspired by the work of E. Izhikevich who, in 2007, published a computational model to solve the distal reward problem using neuromodulation, eligibility traces and spiking neurons with STDP. At that point I had been working more than two years on neuromodulation and reward-based learning. Izhikevich’s work was really inspirational, but in contrast to his position, it was soon clear to me that spiking neurons were not crucial and that something else was driving the learning. This thought hung in the back of my mind for a few years until,  in the summer of 2011, I met Paul Tonelli, a young researcher working on the evolution of plasticity and learning. We discussed the topic of the distal reward learning in the noisy setting of a Dublin’s pub. The suspicion that rare events (or correlations) were a pivotal part of the learning led me to prove it. A week later I had the first version of the algorithm. All simulations and paper were ready in December 2011. Then the long negotiations with the reviewers started and the paper is schedule appear in April 2013 in Neural Computation.